The present invention is generally directed to the management of very large file systems. Such file systems are typically used in conjunction with large data processing systems. More particularly, the present invention is directed to a method and system for rapidly restoring a data file system such as might be present in a Storage Area Network (SAN) environment. Such environments are often used in conjunction with distributed data processing systems. However, the present invention is not limited to these environments.
The advent of SAN file systems and low cost storage have created the possibility of very large file systems with a very large numbers of files. In fact, file systems having capacities of several terabyte (1 terabyte=1012 bytes=1,000 gigabytes) exist today and larger ones are possible. Unfortunately, the time to completely restore a file system from a backup copy is proportional to the file system size. Assuming 20 megabyte/second tape speeds and ignoring tape handling time and any software overhead, the restoration of 1 terabyte of data from tape would require over 10 hours. The customer has the choice of manually restoring files of particular interest first and allowing access to the file system or delaying access until all data has been restored. The first option requires that the system administrator be able to first precisely identify the desired data, since applications that access unrestored data would otherwise fail. The second option involves the delay of restoration of business until all of the files have been restored from tape. Neither of these alternatives is wholly satisfactory.
The present invention employs data structures which are also found within a related invention presented in application Ser. No. 10602156 titled “Parallel High Speed Backup for a Storage Area Network (SAN) File System,” which is being filed concurrently herewith. The present invention also employs techniques found within the parallel support aspects of Hierarchical Storage management features contained within GPFS (General Parallel File System), a product sold by International Business Machines Corporation, the assignee of the present invention, in support of its pSeries product line. However, the present invention also adds new features which provide the capability of accessing backed data on an on-demand basis while still continuing a normal restoration of the file system. The present invention allows applications (that is, application programs) to begin using the file system even immediately after a file system restoration process has just begun, thereby greatly reducing the time the file system and a customer's data is unavailable.
For a better understanding of the environment in which the present invention is employed, the following terms are employed in the art to refer to generally well understood concepts. The definitions provided below are supplied for convenience and for improved understanding of the problems involved and the solution proposed and are not intended as implying variations from generally understood meanings, as appreciated by those skilled in the file system arts. Since the present invention is closely involved with the concepts surrounding files and file systems, it is therefore useful to provide the reader with a brief description of at least some of the more pertinent terms. A more complete list is found in U.S. Pat. No. 6,032,216 which is assigned to the same assignee as the present invention. This patent is hereby incorporated herein by reference. The following glossary of select terms from this patent is provided below since these terms are the ones that are most relevant in terms of providing a better appreciation and understanding of the present invention:
Data/File System Data: These are arbitrary strings of bits which have meaning only in the context of a specific application.
File: A named string of bits which can be accessed by a computer application. A file has certain standard attributes such as length, a modification time and a time of last access.
Metadata: These are the control structures created by the file system software to describe the structure of a file and the use of the disks which contain the file system. Specific types of metadata which apply to file systems of this type are more particularly characterized below and include directories, inodes, allocation maps and logs.
Directories: these are control structures which associate a name with a set of data represented by an inode.
Inode: a data structure which contains the attributes of the file plus a series of pointers to areas of disk (or other storage media) which contain the data which make up the file. An inode may be supplemented by indirect blocks which supplement the inode with additional pointers, say, if the file is large.
Allocation maps: these are control structures which indicate whether specific areas of the disk (or other control structures such as inodes) are in use or are available. This allows software to effectively assign available blocks and inodes to new files. This term is useful for a general understanding of file system operation, but is only peripherally involved with the operation of the present invention.
Logs: these are a set of records used to keep the other types of metadata in synchronization (that is, in consistent states) to guard against loss in failure situations. Logs contain single records which describe related updates to multiple structures. This term is also only peripherally useful, but is provided in the context of alternate solutions as described above.
File system: a software component which manages a defined set of disks (or other media) and provides access to data in ways to facilitate consistent addition, modification and deletion of data and data files. The term is also used to describe the set of data and metadata contained within a specific set of disks (or other media). While the present invention is typically used most frequently in conjunction with rotating magnetic disk storage systems, it is usable with any data storage medium which is capable of being accessed by name with data located in nonadjacent blocks; accordingly, where the terms “disk” or “disk storage” or the like are employed herein, this more general characterization of the storage medium is intended.
Snapshot: a file or set of files that capture the state of the file system at a given point in time.
Metadata controller: a node or processor in a networked computer system (such as the pSeries of scalable parallel systems offered by the assignee of the present invention) through which all access requests to a file are processed. This term is provided for completeness, but is not relevant to an understanding of the operation of the present invention.